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InfoSeer Inc. Response to RIAA/IFPI Request for Information 
On Audio Fingerprinting Technologies 
July 2001 

1 Introduction 

InfoSeer Inc., (the Company), is engaged in the development of digital file identification 
and related technologies, including those for audio files. This document responds to the 
Request for Information (RFI), issued by the Recording Industry Association of America 
(RIAA) and the International Federation of the Phonographic Industry (IFPI) and, step by 
step, attempts to answer all the points raised in the RFL It also expands on explaining the 
surrounding technologies, including distribution control and Peer-to-Peer (P2P) 
commerce, that the Company has developed, or in the case of the latter, is developing. 

The Company's technology, as correctly indicated in the RFI, operates on the actual file 
content and does not alter the file header or the content in any way. Therefore, there are 
no audibility issues; neither can the files have the fingerprint removed, as they exist only 
in the Company's secure database. There is no normal access to that database. The audio 
fingerprinting method, which has unique properties, accuracy and special facilities, and 
the associated systems, (to be described), are fiiUy developed and are currently 
operational, portable and demonstrable. 

The architectures of the system and sub-systems are created in such a way that allows 
scalability and versatility so that they can incorporate new audio technologies when they 
are developed and come into widespread use in the fixture. Furthermore operating 
parameters can be adjusted in software, without returning to file sources, so that 
customization for particular applications is straight forward, and does not need extensive 
re-work of the programs or databases. Therefore the typical possible applications for the 
technology, as described in the RFI, are simple to implement. These points will be 
explained in detail in the appropriate section(s) later. 

The Company's total system is agnostic to, and can operate with, other technologies such 
as Digital Right's Management (DRM) and watermarking. 

This activity by the Company arises because of the demand for the control of Intellectual 
Property (IP) and the associated privacy issues that have been stated by the banking, 
health, federal, defense, movie, publishing and other industries. 

It is fostered by the need for Internet Intellectual Property policies that are a major 
concern of governments worldwide, as exemplified by the Digital Millennium Copyright 
Act (DMCA), its critics, and other efforts in the United States (US) and European 
Commission (EC), amongst others. 

The Company is well positioned to address these issues, as many of the staff have 
previously worked for the Federal Bureau Investigative (FBI), Central Intelligence 
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Agency (CIA), National Security Agency (NSA) and other organizations focused on 
solving the problem of implementing the highest possible levels of privacy and security. 
That is one reason why this Company has developed the philosophy and belief that the 
fingerprinting of files is only one step in the need to protect, and where appropriate 
particularly for the entertainment industries value add, the proprietary information for the 
creator or owner of that information. The other issues, and some solutions, outside the 
direct scope of this RFI, will nevertheless be explained in the appropriate general sections 
below. 

But first, the Company will respond to the audio fingerprinting questions directly raised 
by the RFI. 

2 Logistics 

The Company intends to comply with the logistics requirements. 

3 Reference Architecture 

The Company agrees with and complies with the reference architecture insofar as it 
concems the tracking of fingerprints, metadata and file verification core technology 
methods. However it will be seen that there are several associated "core" technologies 
that the Company uses that enhance this reference model. As stated in the RFI there are 
also applications that require additional or modified architectures. Enhanced 
architectures will be described later in this response. 

4 Application Scenarios 

All the application scenarios stated in the RFI are covered by the technology. Further, 
the Intemet is being "crawled" by "Info Watch" software (referred to in the Company as a 
data collector), on a multi-thread basis and about 450,000 results have been obtained in 
24 hours using just one Tl connection. 

Clearly, this can be fiirther scaled up by duplication of the data collectors and links. 

Cease and desist letters are produced automatically with date and time stamp, and there is 
the facility to let the customer see, check and approve and, if desired, send the letters to 
the appropriate authority (usually an Intemet Service Provider (ISP), electronically. 
Furthermore, the most offered tracks, or specified artists, are inserted into the letters for 
the given unauthorized address without human intervention. More details can be given at 
a later time. 

There are several existing audio tracking services for broadcast applications that are 
analog, for example BDS, a VNU company, headquartered in White Plains, New York. 
The Company's technology will be able to track broadcasts more accurately over the 
Intemet because of the inherent accuracy of digital transmissions. Furthermore, with 
"simulcasters" the technology will be able to give near 100% accuracy rather than the 6 
hours or so per week sampling methods employed by others. (They sample content at 
approx. 4% of the total time). 
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4.1 Audio Content Tracking and Reporting 

a) The Company is already monitoring and compiling reports and charts of Internet 
P2P usage on Napigator, Gnutella, Bearshare, and File Transfer Protocol (FTP), 
sites. These are being used by several organizations. 

b) Airplay/netplay monitoring and charts are not being issued at this time, it is a 
simple matter to organize however and the Company's particular interest is to see 
if webcasters are complying with the DMCA rules governing the frequency 
transmission of a given song in a specified period 

4.2 Internet Audio Content Services 

The Company has created an additional database that is associated with the fingerprint 
and meta-data 

Database shown in the reference architecture. This database contains authorization 
"Rules" and can be dynamically updated with the content owner's intentions. It uses 
information concerning track identities, hitemet Protocol addresses and port numbers and 
if necessary, the whereabouts of the relevant files. 

4.3 Anti-Piracy Investigation and Enforcement 

The RIAA is in possession of InfoSeer's system for anti-piracy and CDR activities. It 
basically works by checking the "fingerprint" database when a suspect CD or CDR is 
played in a coupled computer and verifies whether the sample CDR is known as a 
member's recording or not. Thus, a suspect pirated object can have the tracks 
authenticated. Clearly, the system can be used to authenticate master recordings at CD 
plants and for repertoire analysis and Internet authentication, which is also currently 
enabled and in use. 

a) Suspect recordings are being verified 

b) Repertoire is being analyzed and identified 

c) Masters can be screened 

d) Internet transmissions are being identified 

4.4 Value Added Services 

The Company does not have extensive databases about the ancillary or meta-data 
concerning tracking intelligence. It relies on the many other such databases that exist in 
the market places. Such as the RIAA's, Gracenote's, Muze, Soundscan and others from 
the Labels. Our purpose is to definitively identify the content and relate its accuracy to 
existing available knowledge about the content. (With technology that can do something 
about it). 

a) The Company requires access to external databases to provide meta-data 
after a track has been identified using the fingerprint and associated title 
and artist. 
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b) A major development that is underway in the Company is to enable the 
conmiercial monitization of streaming and downloads of content with, 
promotion and other services. Systems built by the Company can also be 
organized to insert advertising, hot links etc. into a comprehensive 
infrastructure. The system prevents unauthorized transfer of legally 
obtained content, but at the same time allows and monetizes P2P transfers 
so that value added services can be offered. These value added services 
include, guaranteed file quality and download speed, multiple price points, 
line busy indications and availability, facilities that will encourage the 
users to use the service. The resulting transaction analysis and payments 
can be apportioned to copyright holders and artists in a completely 
automatic way. These technologies are discussed later in the next 
sections. 

c) Special promotions and incentives are already built into the overall 
architecture and are operational and demonstrable today. 

5 Technology Documentation Process 

The Company is in total agreement with, welcomes the opportunity to, and will comply 
with the stated phases indicated in points 1 and 2. 

5.1 Phase 1 - Analysis of RFI Responses 
5.1.1 Functional Description. 

The technology takes an integrated spectral analysis with a combination of FFT's and/or 
DCT's spaced at 90 degrees to avoid raised cosine nulls and generates frequency and 
amplitude vectors, ignoring the imaginary component to avoid circumvention by all pass 
group delay and a/d and d/a networks. The obtained vectors are subtracted to give an 
ellipse of uncertainty about each resultant. This important point will be shown to be very 
usefiil later. Many of the most dominant vectors are used in the fingerprint and are 
logged. However, all must not need to be matched. (This is important for certain anti- 
circxmivention measures.) The analysis lasts for 30 seconds but this time is arbitrary and 
in practice has been found suitable for the necessary accuracy. This duration is not 
definitive in that "fingerprints" can also be obtained for less time than this and also 
analysis can occur for the whole music file (or track) where available. 

• The vectors are normalized for amplitude so simple changes in gain are irrelevant. 
They can also be normalized against frequency but this has not been 
implemented, as it has not been found necessary in practice. 

• The content of the database can process the results in a variety of ways using only 
software methods if proved necessary. 
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• The availability of an excess of information about the track enables several anti- 
circumvention facilities to be described later. 

An important point is the ability to adjust the track identification technique in the 
following way. The ratio of false negatives to false positives can be adjusted in software 
without resort to the original music file. This is important for many reasons. The 
Company estimates with the currently adjusted identification criteria that false positives, 
i.e. files that are found to be copyrighted but are actually in the public domain is about 
one in ten bilUon, False negatives, in that those files that should be found as copyrighted 
but are missed is about one in one thousand. As already stated clearly the identity vector 
ellipse for the tracks is adjustable in software and can be made to produce any ratios 
acceptable to the copyright holder and implied legalities. These results are with a 30- 
second analysis. For a three minute song completely analyzed these results would be 
expected to be a factor of about ten higher and lower respectively, (power integration), 
i.e. about one in a 100 biUion false positives and about one in ten thousand false 
negatives. Because the Company does not have an extensive database of fingerprints, 
these estimates have to be proven; currently they are based on the mathematics of our file 
detection and uncertainty criteria coupled with experimental results fi^om about 10,000 
tracks. 

The file E) is under 400 bytes and with "house keeping" (time, date, title, etc.). The total 
is about 1 kByte. Thus, one million songs would need a database of about one-gigabyte; 
this is not a large database to search, which would take approximately one millisecond, 
(or a few microseconds in a parallel search). Because of the need to search rapidly in the 
Company's total infrastructure, short versions of the fingerprint of four bytes are used to 
partition the database so "jump to" commands can be enabled to execute very rapid 
searches. 

5.1.2 Description of the Capabilities of the Technology 

Currently, the fingerprint algorithms run in software at about 27 times real time, 
(including MPS decoding). This means that for a 30 second sample of a file, the 
fingerprint can be derived in a little over one second. The Company has calculated that 
with dedicated DSP's a figure of about 50 times this value is expected. They could also 
be arranged to be scalable and multi-threaded. As explained later in the total system 
architecture, it is expected that one system could simultaneously handle 8,000 real time 
song analyses, i.e. about a T3 total bit-rate (approx. 45Mbits/s), for average good quality 
MP3 files. 

An important point is that originally the Company expected that there would be one 
different fingerprint of a given song for each bit-rate and version or make of MP3 codec. 
This would not slow up the database search, because the file header would enable a 
"jump to" the appropriate section of the database. However, the database would have to 
be correspondingly larger. In practice this has not been found necessary. One fingerprint 
works equally well for the tests we have done on three different most popular Codecs and 
seven MP3 bit-rates from 96kbits/s through 360kbits/s and on up to the CD rate of about 
1.4Mbits/s, with no material difference in detection statistics. This is because of our 
technique of "sounding out" the spectra and dynamics of the spectral content. 



Fingerprints can be derived for MP3 and WMA formats simply by arranging appropriate 
decoding as indicated in the file header. 

An interesting facility is the following: If a track is re-mastered from a given master tape 
and a fingerprint has been established for the first version of the song. 

The Company normally would create two "fingerprints" for such a given (nearly the 
same), track, hideed the original identifier will not verify the identity of the second 
version. However our technology can identify that the second version comes the same 
identical master tape. This is obtainable by our software without re-using the master, just 
by running a program on the track's fingerprints. More detail is explained in the 
following sections. 

5.1.3 Application Scenarios addressed by the Technology 

All applications specified in the RFI are addressed currently by the technology unless 
specifically stated to the contrary. Furthermore many scenarios will be described that are 
not envisaged by the RFI as will become clear below. 

5.1.4 Application Scenarios not Covered by the Technology 

The Company does not know of applications not covered by the technology. To obtain 
fingerprints of all available tracks however, access must be afforded to music tracks and 
meta-databases, as the Company has not populated its own comprehensive independent 
database of tracks and metadata. 

5.1.5 Complementary Technologies Needed 

There are no other technologies needed. However, music and track information is 
required as detailed in the previous paragraph. The described system is built, operative 
and in-use today. 

5.1.6 Optimum Evaluation and Testing 

It is suggested that the system installed at the Anti-Piracy department at the RIAA is used 
for tests, since the application scenarios already described are in action, including the 
web-crawlers for Napigator, Gnutella and FTP sites. Furthermore remote secure access 
to the information is available firom the Company's dedicated (to specific personnel at the 
RIAA) web site, complete with many layouts of reports for the data. Surrounding 
technologies developed by the Company are also installed, to be described below. 

5.1.7 Technology Road Map 

As stated in the introduction, the Company is developing fingerprints for the following 
intellectual properties: 

Movies and TV, Documents, Legal, Health and Banking records. Books, Pictures, CAD 
drawings and JPEGS. 



Each type of fingerprint has different algorithms and requirements. For example, in the 
case of movies, the fingerprint must identify various encoding methods such as DVD, 
MPEG one or two, or identify a movie taken by a camera pointed at a movie screen (that 
may not be horizontally aligned), or may be black and white instead of color. The 
Company has nearly completed this activity and can successfully detect such movie 
conditions. 

Li another example for documents, it is important that the document is the original and 
not accidentally or maliciously altered. (Particularly for bank, health or legal records). 
The fingerprint is robust enough to still identify the original, even if odd paragraphs are 
missing, but also restore the altered document to it's original state unless severely altered, 
in which case the reader is informed of the situation. It is a vital requirement of Top 
Secret Documents for example. This work is finalizing in the next few weeks. 

The Company's core technologies however are not fingerprinting, which is only an 
enabler. They are: 

a) Enabling Super Distribution including P2P, with micro payments and 
various value-add services, and 

b) Control of distribution by router and switch dynamic updates in Litemet or 
Network infirastructures. 

The Super Distribution model is under development and will be demonstrable at the end 
of October this year. This is shown in Appendix A. 

Controlling distribution is fully built and operative today and can be demonstrated live on 
the Litemet; (the system is installed at the RIAA). This is shown in Appendix B. 
Appendix A and B provides the overview diagrams of the architectures. 

Control of content on the Intemet can be accomplished through ISP's and common 
carriers. Control of EP in corporations, universities and agencies can be enabled through 
networks generally, and their vendors, remote offices or embassy's. An important point 
about distribution control is if an attempt is being made to send content to unauthorized 
destinations, a database of "rules" is accessed to ascertain the associated file 
authorization. The rules are set-up by the content owner, and can be dynamically updated 
at will through interfaces to the databases. In private closed networks, intemal addresses 
are used to ensure content can travel to only specified personnel; traffic to the Intemet 
can similarly be regulated. 

The Company can implement the "rules" consistent with the content owner's wishes. 
Furthermore, as well as redirecting the content or discarding it, messages may be 
substituted. For example, in an attempted unauthorized P2P music transaction an audible 
message can be substituted for the music directing the intended recipient to a legally 
obtainable version of the same song. 

Many other marketing activities can be envisaged since the technology is versatile. 



Product Road Map 

a) No software development kits are available. The technology does not, and 
needs not, reside on the desktop for security reasons. There is no general 
or public access to the databases, or the fingerprinting technology. 

b) No third party intellectual property is known, (in good faith), to be 
involved. The Company's core fingerprint and associated technologies are 
believed to be proprietary. 

If customization activities are required, for example in the presentation of reports, 
the Company will undertake this task for the client. However, XML can be used 
to make searches of the databases for offerings on the hitemet. Third party 
Integrators may be used according to client's needs. 

For the purpose of information other conditions may apply to non-audio 
applications that are not the direct concern of this RFL 

5. 1 .9 Intellectual Property 

5.1.10 Circumvention Scenarios 

The Company's philosophy is two fold: 

a) Circumvention Via Methods affecting Audio Quality, These circumvention 
methods would affect audio quality in some way as to render the track xm- 
enjoyable and only at that point is it not identifiable: 

• Cutoff/Reversals, The method is independent of cutoffs at the 
beginning and/or the end of the file and against reversals of file 
transmission. If the file is completely scanned then only 25 seconds of 
the relevant file (whose duration may be 3 to 5 minutes), is needed for 
detection, played forwards or backwards with up to one minute cut- 
offs. 

• Gain Changes. Gain changes are normalized in the fingerprint for 
amplitude and are therefore irrelevant. 

• Frequency Changes. Frequency changes, as opposed to transmission 
speeds, are not found. They would necessitate bit rate converters and 
false file headers. Normalization against such fi"equency changes is 
easy but not currently implemented, as they have not been proven 
necessary. It is likely that such techniques would produce considerable 
sound quality degradation for compressed files. Transmission speeds 
are irrelevant as they are handled simply by accumulating and 
counting packets and samples. 



• Added Noise. Noise has to be added to high levels to defeat the 
method, since the frequency bin size is 2.4 Hz and integration is used. 
Such noise levels would destroy the aural enjoyment of the music. 

• Group Delay Variations, Group delay variations (a/d, d/a or all-pass 
networks) are irrelevant to the methodology and tracks are therefore 
easily detected. (As already mentioned, only the real components of 
vectors are used in the fingerprint.) 

• Re-mastered. Re-mastered equalization fi-om one given master 
requires a new fingerprint. Therefore, it may be several fingerprints, 
all of which identify the same song. However, by taking the second 
differential of the integrated FFT and InfoSeer's other math 
algorithms, the one original master for several versions can be 
definitively identified. It employs the technique of identifying the 
music dynamics that remain essentially unchanged after re-mastering, 
if they originate from the same studio master. (This is because studio 
changes in equalization, even while the track is being played, occur at 
low or subsonic fi-equencies and these are ignored by the fingerprinting 
method.) The Company has enabled master confirmation successfully 
on several re-mastered releases fi"om the same master tape and hence, 
confident of the acoustic principles behind the technique. This is 
important to supplement legal identification and action. 

• New Codecs. When new Codecs are developed in the future only a 
software program is needed to generate a second generation of 
fingerprints to identify the existing tracks in the databases. This 
process therefore will not be manually intensive. 

b) Circumvention methods not widely known. The circumvention method is 
difficult or not widely known, it is therefore, less financially damaging to the 
copyright holder. If it reaches such popularity that legal methods can prevent 
its widespread use, it would also known to the Company that can then employ 
the appropriate and renewable detection and analysis methods. 

The Company, because of the tremendous versatility of hackers, continues to test several 
circumvention scenarios and will continue to test and validate the robustness of the 
technology to a number of common hacker attacks. 

The most important anti-circimivention facility is centered round the technology 
architecture, in that the fingerprinting method is not on the desktop or available to the 
normal user. The client accesses the system to add and modify the rules database and 
they are subject to conventional security techniques. 

A clear circumvention technique is encryption. Encrypted files can be fingerprinted. If 
these are widely distributed then the Company will also be cognizant of them. If not then 



the unauthorized offering and distribution will be on a one or two off basis and therefore 
not be a large loss to the owner of the content. In a private network this information will 
be available or encrypted files can be controlled irrespective of content. 

5.1.11 Intellectual Property Held 

IP is believed to be proprietary particularly concerning total system architecture beyond 
that required by this RFL Several Router and network infi-astructure manufacturers have 
been approached to license the manufacture of hardware for future "content intelligent" 
systems. We would expect that if these negotiations are as successful as they currently 
seem to be then the IP is defendable and enabled. 

5.1.12 Company Details 

5.1.13 Other Information 

The Company is negotiating with several Federal agencies, other IP Associations, 
Corporations, ISP's and Integrators at this time for audio, film, documents and other 
Intellectual Property identification and protection. The control and monetization of 
digital content for the mass market is the most developed at this time and the Company is 
most interested in its facilitation. Therefore we enthusiastically want to pursue this RFI 
for our mutual benefit and would welcome input fi-om the representatives of the music 
industry. 

5.2 Phase 2-Discussion, Demonstration and Testing 

The Company is pleased to respond to any further discussions, clarifications and 
demonstrations indicated in points 1 through 3 of this paragraph. The Company will 
attempt to verify the statements made in this RFI, which have been made in good faith. 

6 Miscellaneous 

6.1 No Obligations 

The no obligation provision is completely understood and agreed with, except those 
obligations concerning non-disclosure undertaken in the NDA document particularly 
about proprietary secrets. The Company similarly undertakes no obUgations in this 
response to the RFI except as provided by the NDA. 

6.2 Non-Discriminatory Policy 

This policy is completely understood and agreed with. While the Company would prefer 
its technology to be recommended and further, used, it understands and agrees with the 
policies that the RIAA and IFPI are acting under and reaUzes the constraints of this 
provision. It is willing to undergo any in depth analysis that will enable the RIAA and 
IFPI to make a meaningful value judgement of the presented technology and system(s). 



6.3 IP Considerations 

The Company welcomes the opportunity to review the report of their own technologies 
so that any omissions or exceptions can be stated and amplified before the report is 
distributed. Such comments will be supplied in a timely manner, after the draft report is 
supplied by the RIAA and IFPI to the Company. The Company welcomes this provision 
and understands that the Associations have their members to protect. 

6.4 Press 

The Company does not wish any press statements to be issued; neither will it issue any 
publicity statement, with out the express written permission or granting of request from 
both parties expressly involved with this RFI. If such permission is granted then both 
parties must agree the text of the press submission before it is transmitted. 
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InfoSeer Private andConfiden^l 



P2P M&ooavment Process 

1 . User A Searches for an Artist. Album or Title. 

2. User A recieves search results. 

3. User A tells User B that he/she wants to download a Song. 

4. Both users connect to the Identification Server. 

5. User B uploads the nnusic to User A through the Identification Server. 

6. The Identification Server recognizes the file using our fingerprinting technology. 

7. User A is charged via nnicropayment system. 
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Appendix B Control of Distribution Arcliitecture 
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